Prevent LoadBalancer updates on follower. #6872
Conversation
tsaarni
requested review from
skriss and
sunjayBhatia
and removed request for
a team
sunjayBhatia
requested review from
a team,
davinci26 and
izturn
and removed request for
a team
Follower instances of Contour do not run loadBalancerStatusWriter and therefore do not read from the channel that receives status updates. The LoadBalancer status updates are still watched and sent to a channel, causing the go routine to block. This led to LoadBalancer updates piling up and consuming memory, eventually causing an out-of-memory condition and killing the Contour process. Signed-off-by: Tero Saarni <[email protected]>
tsaarni
force-pushed
the
lbstatus-on-follower
branch
from
8b2af7d to
920d0ce
Compare
tsaarni
added
the
release-note/small
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #6872 +/- ##
==========================================
- Coverage 81.05% 81.04% -0.02%
==========================================
Files 133 133
Lines 20026 20034 +8
==========================================
+ Hits 16232 16236 +4
- Misses 3500 3504 +4
Partials 294 294
|
| func (s *ServiceStatusLoadBalancerWatcher) notify(lbstatus core_v1.LoadBalancerStatus) { | ||
| s.LBStatus <- lbstatus | ||
| if s.leader.Load() { |
There was a problem hiding this comment.
Is it possible to lose the envoy service event in this way?
There was a problem hiding this comment.
True. Currently the event processing logic relies on Added/Updated/Deleted events alone. It works for other resource types as they will be processed by all Contour instances but I think new approach is needed with the load balancer status updates.
There was a problem hiding this comment.
Maybe we can simply handle it. Notify the latest envoy service when it becomes Leader.
|
The Contour project currently lacks enough contributors to adequately respond to all PRs. This bot triages PRs according to the following rules:
You can:
Please send feedback to the #contour channel in the Kubernetes Slack |
github-actions
bot
added
the
lifecycle/stale
This PR fixes a memory leak that was triggered by LoadBalancer status updates. Only the leader instance runs
loadBalancerStatusWriterand therefore follower does not have anyone reading from the channel that receives status updates. The LoadBalancer status updates are still watched by followers and sent to the channel, causing the go routine callingServiceStatusLoadBalancerWatcher.notify()to block. This led toLoadBalancerStatusupdates piling up and consuming memory, eventually causing an out-of-memory condition and killing the Contour process.Fixes #6860